Assessment and Concept Map Structure: The Interaction Between Subscores and Well-Formed Mental Models

The concept map is a visual tool, a nontraditional assessment tool designed to specify individual mental models through graphic representation. This study investigated relative sensitivity of concept map subscales to well or poorly defined mental models of a targeted domain (learner achievement). Fifty-eight preservice teachers completed a computer-based lesson introducing the educational philosophy concept of functionalism within a traditional society. Analysis of concept maps constructed from learners who completed responses to four essay questions indicated a significant interaction between Subscore and Achievement, F(3,51)=33.42, p< .01, η= .66. The Crosslinks subscale, measuring a learner’s domain integration, is more sensitive than Branches (domain differentiation), Propositions (domain knowledge), or Levels (domain differentiation). Implications for research and practice are discussed. As expert advisory bodies like the United States National Research Council urge policy makers, scientists, and educators to recognize the limits of traditional assessment instruments (2001), concept maps provide an attractive alternative. A concept map is a graphic representation of perceptions of a domain’s relational structure. Each concept map is constructed to visually characterize understanding of the relationships between a targeted domain’s concepts and its subconcepts. Thus, the concept map is inherently applicable to the measurement of higher-order conceptual understanding that characterizes meaningful learning (Ausubel, 1962; Novak & Gowin, 1984; Wiggins & McTighe, 2001). In its broadest application, alternative assessment initiates a process through which instructional goals are explicitly and holistically reinforced within a learning community (Hernandez & Reese, 2004; Wiggins & McTighe, 2001). This requires ongoing feedback and learner growth within an environment of publicly advertised standards and criteria (Hernandez & Reese, 2004; Wiggins & McTighe, 2001). Because a concept map is an explicit representation, it provides an object that can be objectively shared, digested, and discussed. As concrete objects of shared discourse, concept maps can be used as formative assessment tools to shape a learning community’s progress toward instructional goals that require higher-order understanding. Since Joseph D. Novak and his colleagues (Novak, 1990, 1992; Novak & Gowin, 1984; Novak & Musonda, 1991) developed, pioneered, and validated the use of the concept map, educators and researchers have added it to the more traditional array of pedagogical and assessment tools. Assessment and evaluation of a concept map have involved quantification of three characteristics: (a) amount of knowledge, (b) domain differentiation, and (c) domain integration. These parameters are normally measured through subscale scores. Unfortunately, evaluators typically develop idiosyncratic sets of subscales and subscale weightings for each concept map application (Edmondson, 2000; Markham & Mintzes, 1994; Novak, 1990; Novak & Gowin, 1984; Wallace & Mintzes, 1990). This practice curtails generalization across domains and populations, limiting the contribution of concept mapping results within curricular and research efforts (Edmondson, 2000; Reese, 2003a; Shavelson & Ruiz-Primo, 2000). In addition to the generalization issues, ill-conceived scoring rubrics may unintentionally reinforce superficial rather than deeply structured knowledge organization. The National Research Council (2001) has suggested that three key elements should underlie any assessment: (a) cognition: how students learn and organize a targeted domain, (b) observation: how evaluators collect evidence of student learning in a domain, and (c) interpretation: how evaluators conduct an analysis and explanation of those observations. One implication of this “assessment triangle” is that an expert-novice paradigm comparing a learner’s concept map to the expert’s map of a targeted domain might more closely align with the triangle’s cognition vertex than a summation of subscales measuring a concept map’s open-ended richness. Comparisons between expert and novice concept maps are a relatively new area of research, requiring an algorithm and computer program to quantify the maps (Fisher, 2000; Reese, October, 2003; Clariana, Koul, & Salehi, in press; Poindexter & Clariana, in press). More commonly, researchers and educators follow Novak’s paradigm and score concept maps by the sum of 1 Paper presented at the 2004 meeting of the American Educational Research Association, San Diego. © 2004 Wheeling Jesuit University/Center for Educational Technologies/ All rights reserved.